Implement column projection #1443
Conversation
…tion logic to helper method, changed test to use high-level table scan
tests/io/test_pyarrow.py
Outdated
| partition_spec = PartitionSpec( | ||
| PartitionField(2, 1000, VoidTransform(), "void_partition_id"), | ||
| PartitionField(2, 1001, IdentityTransform(), "partition_id"), | ||
| ) |
There was a problem hiding this comment.
i think we'd want to test multiple IdentityTransforms here.
im thinking about a case for multiple-level of partitioning in hive-style.
s3://my_table/a=100/b=foo/...parquet
i think _get_column_projection_values might not support this right now
There was a problem hiding this comment.
hmm, got it, I think it is supported with this new commit, what is doing is that before injecting the value in the RecordBatch, it checks if that name is present in the schema before injecting it.
There was a problem hiding this comment.
Looks like CI caught an interesting case where a new identity partition is added after data files were written. The accessor then cannot find the proper partition record... We need to do something like this
| ) | ||
|
|
||
| partition_spec = PartitionSpec( | ||
| PartitionField(2, 1000, IdentityTransform(), "void_partition_id"), |
There was a problem hiding this comment.
nit: avoid using void since its a type of transform https://iceberg.apache.org/spec/#partition-transforms
| partition_id: int64 | ||
| ---- | ||
| other_field: [["foo"]] | ||
| partition_id: [[1]]""" |
There was a problem hiding this comment.
shouldnt this project void_partition_id=12 as well?
| ) | ||
|
|
||
|
|
||
| def test_identity_transform_columns_projection(tmp_path: str, catalog: InMemoryCatalog) -> None: |
There was a problem hiding this comment.
i was thinking we can test something like 3 fields where 2 are identity partitions.
to check the scenario for multi-level hive partition, for example s3://foo/year=2025/month=06/blah.parquet
This is a fix for issue #1401. In which table scans needed to infer partition column by following the column projection rules
Fixes #1401